Skip to content

POCO serialization#284

Open
cmettler wants to merge 9 commits into
apache:mainfrom
cmettler:feature/serializer
Open

POCO serialization#284
cmettler wants to merge 9 commits into
apache:mainfrom
cmettler:feature/serializer

Conversation

@cmettler

Copy link
Copy Markdown
Contributor

Resolves #186.

I needed Arrow POCO serialization for an internal cross-language interop project (C# ↔ vgi-rpc-python). I took inspiration from System.Text.Json's source generator and MessagePack-CSharp's attribute model, iterated on it with Claude as a coding assistant, and arrived at this implementation.

Figured it might be useful upstream — please take a look and let me know what you think.

See README.md for full documentation and examples.

@CurtHagenlocher CurtHagenlocher requested a review from Copilot March 14, 2026 23:03
@CurtHagenlocher

Copy link
Copy Markdown
Contributor

Sorry, I created a merge conflict by checking in the Parquet variant projects :(.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new POCO serialization subsystem for Apache Arrow in .NET, resolving issue #186. It provides two serialization paths: a source-generator-based AOT-safe approach using [ArrowSerializable] attributes and a reflection-based RecordBatchBuilder for anonymous types/prototyping.

Changes:

  • New Apache.Arrow.Serialization runtime library with attributes, interfaces (IArrowSerializer<T>, IArrowConverter<T>), helper classes, reflection-based RecordBatchBuilder, and extension methods for Arrow IPC serialization
  • New Apache.Arrow.Serialization.Generator Roslyn incremental source generator that emits schema derivation, serialization, deserialization code including polymorphic type support and JSON schema emission
  • Comprehensive test suite covering primitives, collections, nested types, enums, polymorphism, custom converters, callbacks, datetime types, diagnostics, and the reflection-based builder

Reviewed changes

Copilot reviewed 19 out of 20 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
src/Apache.Arrow.Serialization/Attributes.cs Defines all serialization attributes (ArrowSerializable, ArrowField, ArrowType, ArrowIgnore, ArrowMetadata, ArrowPolymorphic, ArrowDerivedType) and callback interface
src/Apache.Arrow.Serialization/IArrowSerializer.cs IArrowSerializer<T> interface with static abstract members and IArrowConverter<T> for custom converters
src/Apache.Arrow.Serialization/ArrowArrayHelper.cs Utility methods for building null arrays, Guid/TimeOnly/TimeSpan/Decimal arrays, and DateTime normalization
src/Apache.Arrow.Serialization/ArrowSerializerExtensions.cs Extension methods for IPC byte/stream serialization and collection convenience methods
src/Apache.Arrow.Serialization/RecordBatchBuilder.cs Reflection-based serializer for anonymous types and non-attributed objects
src/Apache.Arrow.Serialization/README.md Comprehensive documentation covering all features
src/Apache.Arrow.Serialization/Apache.Arrow.Serialization.csproj Runtime library project (net8.0)
src/Apache.Arrow.Serialization.Generator/ArrowSerializerGenerator.cs Main incremental generator: type analysis, diagnostics, and orchestration
src/Apache.Arrow.Serialization.Generator/CodeEmitter.cs Emits serialization/deserialization code for [ArrowSerializable] types
src/Apache.Arrow.Serialization.Generator/PolymorphicCodeEmitter.cs Emits code for [ArrowPolymorphic] type hierarchies
src/Apache.Arrow.Serialization.Generator/JsonSchemaEmitter.cs Emits optional JSON schema descriptors
src/Apache.Arrow.Serialization.Generator/Models.cs Internal model classes for the generator pipeline
src/Apache.Arrow.Serialization.Generator/Apache.Arrow.Serialization.Generator.csproj Generator project (netstandard2.0)
test/Apache.Arrow.Serialization.Tests/SerializationTests.cs Tests for source-generated serialization round-trips
test/Apache.Arrow.Serialization.Tests/RecordBatchBuilderTests.cs Tests for reflection-based builder
test/Apache.Arrow.Serialization.Tests/DiagnosticTests.cs Tests for generator diagnostic reporting
test/Apache.Arrow.Serialization.Tests/TestTypes.cs Shared test type definitions
test/Apache.Arrow.Serialization.Tests/Apache.Arrow.Serialization.Tests.csproj Test project
Directory.Packages.props Adds CodeAnalysis package versions
Apache.Arrow.sln Adds new projects to solution

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment thread Directory.Packages.props Outdated
Comment on lines 35 to 37
<PackageVersion Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.4" />
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp" Version="4.11.0" />
<PackageVersion Include="Microsoft.Bcl.AsyncInterfaces" Version="8.0.0" />
public void Append(object? value)
{
if (value is null) _b.AppendNull();
else _b.Append(new DateTimeOffset((DateTime)value, TimeSpan.Zero));
Comment on lines +620 to +623
// For null slots, we need a stand-in value (first non-null item)
object? standIn = _items.FirstOrDefault(v => v is not null);
foreach (var item in _items)
typedList.Add(item ?? standIn!);
Comment on lines +456 to +461
Line($"if ({access} is {{ }} v_{index}) {{ bld_{index}_idx.Append((short)bld_{index}_dict.Count); bld_{index}_dict.Add(v_{index}.ToString()); }} else bld_{index}_idx.AppendNull();");
else
{
Line($"bld_{index}_idx.Append((short)bld_{index}_dict.Count);");
Line($"bld_{index}_dict.Add({access}.ToString());");
}
Comment on lines +465 to +466
// fall back to AppendNull for now — these are less common in polymorphic scenarios
Line($"bld_{index}.AppendNull(); // TODO: complex type {prop.Type.Kind}");
break;
}
default:
Line($"object? prop_{propIndex} = null; // TODO: unsupported type {prop.Type.Kind}");
Comment on lines +83 to +84
// Remove trailing newline, add comma
sb.Length -= sb.ToString().EndsWith("\r\n") ? 2 : 1;
<TargetFramework>net8.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<Description>Source-generated Apache Arrow serialization for .NET. Provides [ArrowSerializable] attribute and IArrowSerializer&lt;T&gt; interface for compile-time Arrow schema derivation, serialization, and deserialization.</Description>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't need to worry about net6.0 as it's out of support and we'll probably remove it as a build target after the next release. The inability to use with net472 or netstandard2.0 is a greater loss and it might be worth a quick test to see how hard it would be to add support for those.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review! I'll take a look at your comments over the next few days and follow up.

@CurtHagenlocher CurtHagenlocher left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is great and has long been missing.

Can you please fix the merge conflict and the white space that our linter doesn't like? I think the Documentation task failure can be addressed by editing ci/scripts/docs.sh and doing something like

pushd "${source_dir}/src/Apache.Arrow.Serialization"
dotnet build -c Release
popd

before trying to build the documentation.

We should probably also add validation for the release by adding something to dev/release/verify_rc.sh like

reference_package "Apache.Arrow.Serialization" "Apache.Arrow.Serialization.Tests"

but I'm not sure the reference_package will handle the PrivateAssets="all" in the project file. If it doesn't, we could consider figuring that out later after the bulk of the change is checked in.


namespace Apache.Arrow.Serialization.Tests;

public class DiagnosticTests

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice tests!

Comment thread Apache.Arrow.sln
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Debug|x64 = Debug|x64

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to avoid adding all these targets. Is there something bitness-specific in these changes?

{
var list = items as IReadOnlyList<T> ?? items.ToList();
if (list.Count == 0)
throw new ArgumentException("Cannot infer schema from empty collection.", nameof(items));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really true though? We use the type to infer the schema, not the data. It would be annoying for someone to have to special case an empty list if they want to serialize it.

builders.Add(CreateColumnBuilder(propType, arrowType));
}

var schema = new Schema.Builder();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider moving schema above the foreach and adding the fields directly into the schema builder instead of a temporary list.

<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would it take to make this work for .NET 4.7.2? Is that even plausible?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the library now targets netstandard2.0;net462;net8.0 (matching Apache.Arrow), with tests passing on both net8.0 (197) and net472 (182 — the gap is only DateOnly/TimeOnly/Half, which don't exist there). To make that possible, IArrowSerializer changed from static abstract members (which require the .NET 8+ runtime) to an instance interface resolved through a generated, module-initializer-populated registry — the MessagePack IMessagePackFormatter pattern — so the public extension API is identical on every TFM, still reflection-free and AOT-safe.

| `float` | `Float32` | |
| `double` | `Float64` | |
| `Half` | `Float16` | |
| `decimal` | `Decimal128(38, 18)` | Configurable via `[ArrowType("decimal128(28, 10)")]` |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be worth pointing out in the documentation that a CLR decimal is not a perfect match for an Arrow decimal.


namespace Apache.Arrow.Serialization.Generator
{
internal enum TypeKind2

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider a more descriptive name. What about ArrowTypeKind?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All open issues should be addressed.

Variant integration — separate PR or fold in here?

Since Variant landed in Apache.Arrow, I've prototyped a full serializer support for it on a local branch stacked on this one (~9 commits, 231 tests on net8.0 / 216 on net462. Before pushing anything: would you prefer this as a follow-up PR once this one merges, or folded into this PR? My instinct is follow-up — this PR is large enough — but happy either way.

A quick tour, in three layers:

1. VariantValue properties map to the Variant extension type — a column that holds any shape per row:

using Apache.Arrow.Scalars.Variant;

[ArrowSerializable]
public partial record Event
{
    public string Name { get; init; } = "";
    public VariantValue Payload { get; init; }
}

var evt = new Event
{
    Name = "click",
    Payload = VariantValue.FromObject(new Dictionary<string, VariantValue>
    {
        ["x"]    = VariantValue.FromInt32(10),
        ["tags"] = VariantValue.FromArray(VariantValue.FromString("a"), VariantValue.FromString("b")),
    }),
};

var restored = Event.FromRecordBatch(Event.ToRecordBatch(evt));
int x = restored.Payload.AsObject()["x"].AsInt32();   // IPC round-trips too

2. [ArrowType("variant")] stores any POCO/collection property as a self-describing Variant column instead of Struct/List — fully source-generated, recursive, no reflection:

[ArrowSerializable]
public partial record Order
{
    public Address Shipping { get; init; } = new();      // Struct column (default)

    [ArrowType("variant")]
    public Address? Billing { get; init; }                // Variant column

    [ArrowType("variant")]
    public Dictionary<string, double> Prices { get; init; } = new();
}

On read, missing fields default and unknown fields are ignored — old readers survive new writers and vice versa.

3. An alternative wire format for [ArrowPolymorphic] — instead of the flat discriminator + union-of-all-fields schema, one Variant column with self-describing rows:

[ArrowPolymorphic(Format = ArrowPolymorphicFormat.Variant)]
[ArrowDerivedType(typeof(Order), "order")]
[ArrowDerivedType(typeof(Ping), "ping")]
public abstract partial record Message;

// wire:
//   {"$type":"order","Id":"A1","Shipping":{"City":"Berlin"},"Tags":["a","b"]}
//   {"$type":"ping","Seq":7}

The schema never widens as derived types are added, and the flat union's property-type restrictions disappear (nested records, collections, and dictionaries all work). Flat union stays the default — it's the right choice for columnar analytics; the README has a trade-off table. There are also opt-in read policies (MissingFieldHandling / TypeMismatchHandling) for consuming foreign-written batches.

One piece you may want in this PR regardless: while testing IPC round-trips I found that extension columns don't rehydrate after ArrowStreamReaderExtensionTypeRegistry.Default starts empty, so the generated (GuidArray) cast for the existing uuid mapping fails on any deserialized stream. The variant branch fixes this by registering the uuid (and variant) definitions from a module initializer (never overwriting existing registrations), plus a Guid IPC regression test. Happy to cherry-pick just that fix into this PR now if you'd like.

Christoph Mettler and others added 6 commits July 7, 2026 14:25
Adds Apache.Arrow.Serialization with a Roslyn incremental source generator
that emits compile-time Arrow schema derivation, serialization, and
deserialization for types marked with [ArrowSerializable].

- Runtime library (Apache.Arrow.Serialization): attributes, helpers,
  IPC extension methods, reflection-based RecordBatchBuilder
- Source generator (Apache.Arrow.Serialization.Generator): code emission
  for 31+ type mappings, polymorphism, custom converters, callbacks
- Test suite: 197 tests covering all supported types and features
- Integrated into solution, central package management, Apache 2.0 headers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Align with upstream CI which uses .NET 8.0 SDK. All 197 tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove src/test solution folders from sln so serialization projects
appear at root level like all other projects. Change serialization
library target from net8.0;net10.0 to net8.0 for CI compatibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename TypeKind2 enum to ArrowTypeKind (review suggestion)
- RecordBatchBuilder: allow empty collections (schema is inferred from
  the type, not the data) and add fields directly to Schema.Builder
- RecordBatchBuilder: convert non-UTC DateTime kinds via
  ToUtcDateTimeOffset instead of misinterpreting local time as UTC
- RecordBatchBuilder: handle all-null struct columns by creating a
  default stand-in instance instead of NullReferenceException
- PolymorphicCodeEmitter: deduplicate enum dictionary entries so
  repeated values share one index
- PolymorphicCodeEmitter: throw NotSupportedException for unsupported
  complex property types instead of silently serializing null
- Pack the source generator into the Apache.Arrow.Serialization nupkg
  (analyzers/dotnet/cs) and add the package README
- docs.sh: build the serialization library before docfx metadata
- verify_rc.sh: verify the Apache.Arrow.Serialization package
- Directory.Packages.props: restore alphabetical package ordering
- README: document CLR decimal vs Arrow Decimal128 mismatch
- Run dotnet format over the serialization projects

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cmettler cmettler force-pushed the feature/serializer branch from c6c05ba to 922cb4a Compare July 7, 2026 12:43
Christoph Mettler and others added 3 commits July 7, 2026 18:45
Multi-target Apache.Arrow.Serialization as netstandard2.0;net462;net8.0
(matching Apache.Arrow) so the serializer works on .NET Framework 4.6.2+
including 4.7.2.

The static-abstract IArrowSerializer<T> interface required .NET 7+ runtime
support and could not be polyfilled. It is redefined as an instance
interface (the resolver pattern used by MessagePack-CSharp and
System.Text.Json): the generator now emits, per type, a serializer class
delegating to the generated statics and registers it with the new
ArrowSerializerRegistry via [ModuleInitializer]. Resolution is one
static-generic field read; no reflection, still AOT-safe. The generated
statics on each type are unchanged and remain directly usable.

- Add IArrowSerializable marker, implemented by generated partials; the
  generic extension methods keep their exact signatures on all TFMs with
  a `where T : IArrowSerializable` constraint and registry dispatch
- Generator emits an internal ModuleInitializerAttribute polyfill when the
  consumer compilation lacks an accessible one (referenced assemblies such
  as Microsoft.CodeAnalysis carry inaccessible internal copies, so the
  probe checks accessibility, not mere existence)
- Replace emitted Enum.Parse<T> (missing on .NET Framework) with an
  ArrowArrayHelper.ParseEnum<T> helper
- Guard DateOnly/TimeOnly/Half code paths with NET6/NET5 conditionals,
  following the same pattern as Apache.Arrow itself; polyfill
  RequiresUnreferencedCode on downlevel TFMs
- Tests multi-target net8.0;net472 on Windows: 197 pass on net8.0,
  182 on net472 (DateOnly/TimeOnly/Half tests are #if-gated)
- README: document target frameworks, .NET Framework prerequisites, and
  the registry-based dispatch

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The RC Verify job iterates every directory under src/ and fetches a
matching nupkg; Apache.Arrow.Serialization.Generator is not packable
(it ships inside the Apache.Arrow.Serialization package as an analyzer),
so skip it in the artifact fetch loop.

Add Apache.Arrow.Serialization.Tests to Apache.Arrow.Tests.slnf so the
CI test job and RC verification actually run the serialization tests.
During binary verification the generator project reference is dropped in
favor of the packaged analyzer, and the generator unit tests
(DiagnosticTests) are removed there because they need the generator
assembly as a compile-time reference, which a packaged analyzer cannot
provide; they still run against the source tree.

Include the Apache.Arrow.Serialization README in the docfx site as a
conceptual "Serialization" page (docs/_site/serialization/) with a toc
entry and a link from the landing page. Verified locally with
"docfx metadata/build --warningsAsErrors": 0 warnings, 0 errors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docfx metadata loads projects with Configuration=Release, so the source
generator referenced as an analyzer only resolves from the Release output
path. The plain (Debug) build left the analyzer unresolved, producing a
FailedToResolveAnalyzer warning that --warningsAsErrors turned into the
Documentation job failure (docfx exit code 255).

Reproduced in a fresh clone: cold Debug build fails metadata with the
warning; cold Release build passes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@CurtHagenlocher

Copy link
Copy Markdown
Contributor

My instinct is follow-up — this PR is large enough — but happy either way.

I think that makes sense.

extension columns don't rehydrate after ArrowStreamReader

Can you clarify this with a specific code sequence? There are constructors which take an ArrowContext, and my intent was that code opts into extension types during IPC deserialization by supplying one with the expected types configured. (Of course, it's been about four months and the details have long since been GC'd :/.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

How to serialize POCOs to a Table?

3 participants